feat(capture): vertex service-account auth for vision captioning - #3362
feat(capture): vertex service-account auth for vision captioning#3362WaterrrForever wants to merge 2 commits into
Conversation
The Vertex Gemini endpoint rejects plain API keys, so tenants whose Google access is service-account based could never enable vision captioning: the vision phase degraded with provider-error and asset-descriptions.md fell back to filename echoes. Opt in with GOOGLE_GENAI_USE_VERTEXAI=true (the @google/genai SDK convention). Credentials come from GOOGLE_SERVICE_ACCOUNT_INFO as inline SA JSON when set, else ADC; project falls back to the SA's own project_id and location defaults to global. The flag outranks a bare GEMINI_API_KEY because a tenant that sets it is saying its key material is Vertex-side; OPENROUTER_API_KEY stays the top-priority opt-in. Vertex default model is gemini-2.5-flash-lite — Vertex does not publish the 3.x flash-lite preview name the Gemini API path defaults to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vertex serves the GA gemini-3.1-flash-lite name (the -preview alias is Gemini-API-only), so the Vertex path can default to the same tier as the API-key path instead of stepping down to 2.5. Verified multimodal 200 on the worker service account.
0c1c259 to
80b536d
Compare
miguel-heygen
left a comment
miguel-heygen
left a comment
There was a problem hiding this comment.
Reviewed exact head 80b536d21b2b2ba17f34ea45ac05bc54395b5757. All required checks are green; BLOCKED is only the reviewer gate.
Strengths
packages/cli/src/capture/contentExtractor.ts:308-331keeps provider precedence explicit and uses a Vertex-valid default model.- The service-account JSON path is isolated and covered with focused configuration/provider tests.
Blocking issue
- blocker —
packages/cli/src/capture/contentExtractor.ts:247-266,387-395: the documented ADC fallback can return{ project: undefined, location: "global" }, which is then passed tonew GoogleGenAI({ vertexai: true, ... }). In the pinned@google/genaiclient, Vertex auth is considered sufficient only when both project+location (or an API key/custom base URL) exist; ADC credentials alone do not supply the missing project at this constructor boundary, so it throwsAuthentication is not set upbefore captioning.GOOGLE_GENAI_USE_VERTEXAI=truewith ADC and no explicit project therefore degrades tointernal-error, contradicting the advertised fallback. Resolve the ADC project ID before constructing the client, or reject this configuration explicitly with an actionable outcome, and add the no-project ADC regression.
The PR body also still names gemini-2.5-flash-lite, while this exact head correctly uses gemini-3.1-flash-lite; please update the description when fixing the blocker.
Verdict: REQUEST CHANGES
Reasoning: the new ADC-only configuration path cannot initialize the pinned Vertex client unless the caller also supplies an explicit project, so a stated primary use case fails before its first request.
— Magi
|
Closing — superseded by #3561, which is a functional superset of this change across the same three files. Worth recording what #3561 changes beyond this PR, since it is not a pure extension: it swaps the credential contract. This PR used Confirmed independently by Rames and Magi in review of #3561. |
2 participants
Why
The capture pipeline's vision captioning only accepts a Gemini API key (
GEMINI_API_KEY/GOOGLE_API_KEY). Tenants whose Google access is service-account based — HeyGen's own workers included — can never turn it on: the Vertex endpoint rejects plain API keys, and the GCP key present in the worker env is not valid for the Gemini API. Observed effect: every capture'svisionphase degrades withprovider-errorandasset-descriptions.mdfalls back to filename echoes, which downstream agents then compensate for by opening images one by one.What
GOOGLE_GENAI_USE_VERTEXAI=true(the@google/genaiSDK's own convention) routes captioning through Gemini on Vertex AI:GOOGLE_SERVICE_ACCOUNT_INFO(inline SA JSON — the same variable HeyGen's multivendor Gemini provider already reads), else ADCproject_id; location defaults toglobal(GOOGLE_CLOUD_PROJECT/GOOGLE_CLOUD_LOCATIONoverride)OPENROUTER_API_KEY→ Vertex flag →GEMINI_API_KEY. The flag outranks a bare key because a tenant that sets it is saying its key material is Vertex-side.gemini-2.5-flash-lite(Vertex does not publish thegemini-3.1-flash-lite-previewname the API-key path defaults to; verified 404 vs 200).HYPERFRAMES_GEMINI_MODELstill overrides.asset-descriptions.mdheader now reflects the Vertex option.Testing
vitest run src/capture/contentExtractor.test.ts— 18 passed (5 new: config resolution × 4, provider selection through the mocked SDK asserting the Vertex default model).tsc --noEmit— no errors in the touched files.GOOGLE_GENAI_USE_VERTEXAI=true+ the existing worker SA:visionphase completed,3 images captioned with Vertex Gemini, captions are genuine content descriptions (og-image: "A black text logo, ANTHROPIC, is centered on a light background"), 32 SVGs rasterized+captioned.🤖 Generated with Claude Code